Today, I present suggestions for how to calculate averaging weights for a given prediction of the winner of the presidency in each state, and of the percent popular vote in that state. These methods suggestions were inspired by the reporting of Brier scores and other prediction accuracy statistics by Simon Jackman, Sam Wang, and Drew Linzer.
State-level outcomes (thus EV outcomes)
Ferro's adjusted Brier score is best because it accounts for the effects of sample size on the Brier score.
Next, subtract that Brier score from one, which is the highest possible value for a Brier score. The result is an absolute score that increases as the Brier score decreases. Recall that the Brier score is larger when there is greater distance between the predicted and observed values.
Next, we repeat that process for all aggregators that have made predictions at that distance from election day.
Next, we normalize all the absolute scores by the summed absolute scores to give each model a relative weight.
Finally, we weight each model by its relative weight when averaging.
This method could easily be modified to give models weights corresponding to entire prediction histories, and/or to prediction within a given time interval at a given distance from election. It could also be extended to deal with one-off forecasts that are never updated. Because state-level outcomes largely determine the electoral vote, I propose that the same model weight calculated as above could be used when averaging electoral vote distributions.
State-level shares of popular vote
These methods have a lot of nice features:
- They result in weights that are easily interpreted.
- The weights can also be decomposed into different components because they are based on the Brier score and root mean squared error. For example, the Brier score can be decomposed to examine calibration and uncertainty effects. The mean squared error can be decomposed into bias and variance components.
- The methods are flexible enough to accommodate any scope of predictive power that interests researchers.